Skip to content

Add HDF5 Blosc2 filter (id 32026) to libblosc2-jna - #1621

Open
djkwon26s wants to merge 1 commit into
Unidata:maint-5.xfrom
djkwon26s:blosc2-filter
Open

djkwon26s wants to merge 1 commit into
Unidata:maint-5.xfrom
djkwon26s:blosc2-filter

Conversation

@djkwon26s

Copy link
Copy Markdown

Part of #1620 (HDF5 Blosc2 filter, id 32026).

Summary

Adds a decoder for the HDF5 Blosc2 filter (id 32026) to libblosc2-jna, so netCDF-4/HDF5 files written with H5Zblosc2 (hdf5_plugins; also shipped in the Python hdf5plugin package) can be read.

  • Blosc2Filter + Blosc2Filter.Provider, registered in META-INF/services/ucar.nc2.filter.FilterProvider next to the existing BloscFilter (32001). Decode only; encode throws UnsupportedOperationException.
  • LibBlosc2.decodeFrame(byte[] frame, int nbytes) follows the decompression branch of H5Zblosc2.c:
    • each HDF5 chunk is one Blosc2 contiguous frame (blosc2_schunk_from_buffer, no copy);
    • rank >= 2 chunks carry a b2nd metalayer and are padded to the block shape, so they are decoded with b2nd_from_schunk + b2nd_to_cbuffer;
    • rank 1 chunks are decoded with blosc2_schunk_decompress_chunk.
  • The output size comes from cd_values[3] (chunk size in bytes), not the frame header, because the frame size can include padding.
  • Bad input fails with IOException instead of reaching native code: frame magic check, null schunk check, return-code checks, size check.
  • No change to libblosc2-native; the functions used are in the bundled C-Blosc2 2.23.1.

Tests (TestBlosc2Filter)

Test files are small (about 37 KB each) and were made with netCDF-C + the hdf5_plugins Blosc2 filter via ncgen (_Filter = "32026,...", _ChunkSizes); see src/test/data/README.md.

  • provider lookup by id and by name
  • full read of time (rank 1), temp (rank 3, float), count (rank 2, int) compared with an uncompressed copy
  • subset read across partial edge chunks
  • 16 concurrent reads on 4 threads
  • single-chunk decode, rank 3 and rank 1, compared with expected bytes
  • rejects non-frame input, truncated frames, wrong cd_values[3], missing cd_values[3]

./gradlew :libblosc2-jna:spotlessCheck :libblosc2-jna:testWithJdk25 passes locally (Windows x86_64). I did not have a JDK 8 toolchain locally for the default test task.

I also checked this against 8 files from our project: 7 were bit-identical to their uncompressed originals, and the 8th (an operational file, about 650 million values) read completely with no errors.

🤖 Generated with Claude Code

Decode chunks written by the HDF5 Blosc2 filter (H5Zblosc2.c in the HDF
Group hdf5_plugins repository). Each HDF5 chunk is one Blosc2 frame:
chunks of rank >= 2 carry a b2nd metalayer and are decoded with
b2nd_to_cbuffer (which removes the block padding), rank 1 chunks with
blosc2_schunk_decompress_chunk. The decoded size comes from cd_values[3].

- LibBlosc2: add schunk/b2nd bindings and decodeFrame()
- Blosc2Filter + FilterProvider, registered alongside BloscFilter
- Tests with a small netCDF-4 file written by netCDF-C 4.9.3 and the
  HDF5 Blosc2 plugin (rank 1/2/3 variables, bitshuffle/byte shuffle/none,
  zstd/lz4, partial edge chunks), single chunk frames, truncated and
  invalid input, and concurrent reads

Refs Unidata#1620

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Nov 18, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants